Non-product image identification

ABSTRACT

An image recognition technique includes obtaining color characteristic values of pixels in a source image; determining a main color of the source image based at least in part on the color characteristic values; and determining whether the source image is a non-product image based at least in part on the main color of the source image.

CROSS REFERENCE TO OTHER APPLICATIONS

This application claims priority to People's Republic of China PatentApplication No. 200910149552.8 entitled IMAGE RECOGNITION METHOD ANDDEVICE filed Jul. 2, 2009 which is incorporated herein by reference forall purposes.

FIELD OF THE INVENTION

The present application relates to the field of image processing and inparticular to image recognition.

BACKGROUND OF THE INVENTION

The development of multimedia technologies has enriched informationpresentation in software applications. Many image processing techniqueshave been developed for computer applications and especially for theInternet.

Taking an electronic commerce application as an example, existingelectronic commerce systems generally support the function of attachinga picture to a product. A vendor publishing his product offerings overthe network can upload pictures of the product in addition to textualdescriptions. An image can exhibit a product more intuitively than meretext and, in most cases, the picture of a product can also be animportant criterion based upon which a buyer confirms authenticity ofinformation on the product.

For a majority of categories of products, a real photograph of a productshould ideally be used as a picture of the product because the realphotograph shows the appearance and property of the product intuitively.Some vendors, however, often upload irrelevant pictures such as anadvertisement for the vendor, a sale record, a picture that includes auser manual, etc., which do not accurately reflect the properties of theproduct itself. Furthermore, such pictures hinder the efforts of datamining by the website owners and prevent effective categorization ofproducts based on their appearances. Moreover, some website owners maywish to classify images unrelated to the actual products differentlyfrom images depicting products. Techniques for identifying anddistinguishing different types of images are needed.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIGS. 1A and 1B illustrate examples of non-product images.

FIG. 2 is a flowchart illustrating an embodiment of an image recognitionprocess.

FIG. 3 is a flowchart illustrating an embodiment of a process fordetermining whether a source image is a non-product image.

FIG. 4 is a diagram illustrating an example of a source image dividedinto blocks for determining whether the image is a non-product image.

FIG. 5 is a flowchart illustrating an embodiment of a process fordetermining whether a source image is non-product based on its imageblocks.

FIG. 6 is a block diagram illustrating an embodiment of a systemconfigured to recognize non-product images.

FIG. 7 is a block diagram illustrating another embodiment of a systemconfigured to recognize non-product images.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

FIGS. 1A and 1B illustrate examples of non-product images found on anelectronic commerce website. FIG. 1A illustrates an image displaying astore policy and FIG. 1B illustrates a sizing chart image for measuringclothes. These pictures do not depict the product for sale and thereforeare considered as non-product images if included along with informationfor the product. In some existing systems, non-product images arerecognized manually and removed.

In the following specification, an automated image analysis techniquefor identifying non-product images is disclosed. In some embodiments,one or more color features of the image, such as a main color of theimage or main colors of sub-blocks of the image, are obtained and usedto determine whether the image is a non-product image. In someembodiments, one or more rules are applied to the color features to makethe determination.

FIG. 2 is a flowchart illustrating an embodiment of an image recognitionprocess. Process 200 may be performed on a system such as system 600 ofFIG. 6. The process can be described in a general context of computerexecutable instructions, e.g., a program module, executed by a computer.Generally, the program module includes a routine, program, object,component, data structure, etc., which executes a specific task orembodies a specific abstract type of data. The process can also beimplemented in a distributed computing environment in which a task isexecuted by a remote processing device connected via a communicationnetwork. In the distributed computing environment, the program modulecan be located in a local or remote computer storage medium including astorage device.

At 201, color characteristic values of pixels in a source image areobtained. It is assumed that the source image is a color image ratherthan a binary or gray scale image. In some embodiments, the colorcharacteristic values are obtained by reading from the source imagefile. In some embodiments, the color characteristic values are receivedfrom a sender such as an image server used to store the source imagefile. In some embodiments, the source image is optionally converted tobe recognized into a uniform format and scaled into a uniform sizebefore the color characteristic values are acquired.

As used herein, a color model is a mathematical model describing the waycolors can be represented as tuples of numbers, typically as three orfour components. RGB and CMYK are examples of commonly used colormodels. Adding a mapping function between the color model and a certainreference color space results in a definite “footprint” within thereference color space. In a three-dimensional color space, thethree-dimensional coordinate axes correspond to three independent colorparameters so that each color has a corresponding space position andvice versa. A point in the space corresponds to a specific color. Forinstance, when colors are displayed on a computer monitor, they areusually defined in the RGB (red, green, blue) color space. This isanother way of making nearly the same colors (limited by thereproduction medium, such as the phosphor (CRT) or filters and backlight(LCD)), and red, green, blue can be considered as the X, Y, and Z axes.Another way of making the same colors is to use their Hue (X axis),their Saturation (Y axis), and their brightness Value (Z axis). This iscalled the HSV color space. Other color space examples include YUV,YCbCr, etc. Many color spaces can be represented as three-dimensional(X, Y, Z) values in this manner.

RGB-based color model is discussed extensively in the followingdiscussion for purposes of illustration. Other color models can be usedas well. For example, colors black and white are typically representedusing 0 and 1, respectively, in a binary image comprising black andwhite pixels only; and the color characteristic value of a pixel in agrey scale image is presented with a grey scale value. Values of theblack and white colors and gray scale values can be regarded as specialcases of a RGB value with a conversion relationship with the RGB value.Moreover, for a color image, its color can be represented variously asderived from RGB, e.g., a value of YCrCb (Brightness/Hue/Saturation),etc.

At 202, a main color of the source image is determined based at least inpart on the color characteristic values. In some embodiments, for eachcolor characteristic value, the number of pixels corresponding to thecolor characteristic value is counted. The results are compared todetermine the main color of the source image. As used herein, a maincolor refers to a color with the highest frequency of occurring in theimage. In other words, it is the color that corresponds to the colorcharacteristic value that has the highest pixel count.

At 203, it is determined whether the source image is a non-product imagebased at least in part on the magnitude of the color characteristicvalue corresponding to the main color of the source image. In someembodiments, multiple rules are applied to make the determination.Examples of the rules are described in greater detail below.

The non-product image may be handled in various ways. For example, thenon-product image may be automatically deleted from the productinformation with which the image is associated, or a warning may be sentto the user who uploaded the non-product image. In some embodiments,confirmation by a human (such as a system administrator) or by anadditional image processing system is needed before the non-productimage is deleted. In some embodiments, non-product images are classifiedinto a different image category than product images depicting theproduct.

FIG. 3 is a flowchart illustrating an embodiment of a process fordetermining whether a source image is a non-product image. Process 300may be used to implement step 203 of FIG. 2. In the example shown, theprocess applies several rules to make determination of whether a sourceimage is a non-product image.

Initially, a general rule is applied for making a preliminarydetermination based on the determined main color of the source image.Generally, a color source image with a main color other than black orwhite can be considered as a product image, or conversely, a colorsource image with a main color of black or white will have a greatprobability of being a non-product image. Thus, at 302, it is determinedwhether the main color is black or white.

In some embodiments, a color is deemed to be black if it has a colorcharacteristic value that is below a threshold A, and a color is deemedto be white if its color characteristic value that is above a thresholdB. A set of colors with a color characteristic value between A and B arereferred collectively to as “other colors”. A and B take on specificvalues depending upon how to quantize a color space may vary dependingon implementation.

If the main color is neither black nor white, at 304, the source imageis deemed to be a product image. Otherwise, additional rules areapplied. Specifically, at 306, the source image is divided into severalimage blocks and main colors of the respective image blocks aredetermined according to the number of pixels corresponding to respectivecolor characteristic values in the respective image blocks.

At 308, whether the source image is a non-product image is determinedbased at least in part on the main colors of the respective imageblocks. The determination process of a main color of an image block issimilar to that for determining the main color of the source image as awhole. By dividing an image into several blocks and determining thenon-product image status based on the distribution of colors in theimage blocks can more accurately determine whether an image is anon-product image.

For example, when the main color of the source image is black or white,determination can further be performed according to the number of imageblocks with a specific main color and/or the relative position of animage block with a specific main color in the source image.

In this example, the source image is RGB based and there are 8 differentlevels of intensity for each of the three primary colors of R (Red), G(Green) and B (Blue). Thus, the color characteristic value ranges fromRGB000 to RGB777 and the entire color space is quantized into 512colors. For example, RGB235 represents an intensity value 2 for redcolor, an intensity value 3 for green color, and an intensity value 5for blue color.

In this example, some specific colors are defined as follows accordingto the foregoing method for quantizing a color space for laterconvenient descriptions: a color with each of R, G, and B values lessthan or equal to 1 is defined as black; a color with each of R, G and Bvalues greater than or equal to 6 is defined as white. Black and whiteare further specified as follows:

RGB000 represents pure black;

RGB001, RGB010, RGB011, RGB100, RGB101 and RGB110 represent dark black;

RGB666, RGB667, RGB676 and RGB766 represent dark white; moreparticularly, RGB 666 also represents grey white;

RGB777, RGB677, RGB767 and RGB776 represent bright white; and RGB 777also represents pure white.

Based on characteristics of the human visual system, the presentembodiment introduces the concept of a valid color for more reasonableand accurate recognition. If the ratio of the number of pixels withnon-gray color to the number of all the pixels is above a threshold, thecolor can be regarded as a valid color. Valid colors are primarily usedfor measuring the degree to which an image is colorful. As used herein,gray color is defined as a color with equal values of R, G, and B. Ascan be seen, the pure black (RGB000), grey white (RGB666) and pure white(RGB777) defined as above also fall into the category of grey color. Inpractice, a color with equal values of R, G, and B is typically a darkcolor and therefore the present embodiment also takes into account sucha category of color. The threshold is set for the purpose of neglectingsome colors seldom occurring in the image and experimental datademonstrates that the threshold can be set as approximately 5/1000.Other threshold values may be used.

FIG. 4 is a diagram illustrating an example of a source image dividedinto blocks for determining whether the image is a non-product image. Inthis example, the image is divided into 9 image blocks according to a3×3 grid layout. This division ensures that colors on the sides,corners, and the center of the source image are evaluated separately andavoids overly complicated determination rule to improve recognitionspeed. Other divisions are possible in other embodiments.

In this example, the source image is divided into 9 image blocks whichare denoted sequentially with “a block 0”, “a block 1”, . . . “a block8” from the left to the right and the top to the bottom as illustratedin FIG. 4 for convenient descriptions. Main colors of the respectiveimage blocks are determined respectively.

FIG. 5 is a flowchart illustrating an embodiment of a process fordetermining whether a source image is a non-product image based on itsimage blocks. Process 500 may be used to implement 308 of FIG. 3.

At 502, it is determined whether the main color of the source image ispure black, pure white, or something else.

If, at 530, it is determined that the main color of the source image isneither pure black nor pure white, then it is a product image.

If, at 504, the main color of the source image is determined to be pureblack, it is possibly unrelated to the product and it is furtherdetermined whether the main colors of all nine image blocks are black,at 506. If so, the image is deemed to be a non-product image at 508;otherwise, it is deemed to be a product image at 510.

If, at 516, the main color of the source image is determined to be purewhite, it is further determined, at 518, whether all the main colors ofthe nine image blocks are white. If no, the image is deemed to be aproduct image at 508. Otherwise, at 518, it is determined whether thenumber of grey-white blocks is 0, the number of pure white blocks isgreater than 6 and the number of bright white blocks is greater than 7.If so, control is transferred to 520; otherwise, control is transferredto 522.

At 520, it is determined whether block 4 is bright white, and at leastone of the blocks 1, 4 and 7 is pure white. If so, the image is deemedto be a non-product image at 510; otherwise, it is a product image at508.

At 522, it is determined whether the number of dark white blocks is 9and the number of grey white blocks is greater than 6. If so, the imageis a non-product image; otherwise, further determination is performed.At 524, it is determined whether the number of valid colors is less than5, and if yes, the image is a product image; otherwise, it isdetermined, at 526, whether any of the following conditions (a-d) issatisfied. If so, it is a non-product image; otherwise, it is a productimage. Conditions a-d include: (a) the number of bright white blocks is9, the number of pure white blocks is 3, and block 4 is not pure white;(b) the number of pure white blocks is more than 7, block 4 is purewhite, and blocks 1, 3, 5 and 7 are not grey white; (c) the number ofvalid colors is 1, the number of pure white blocks is 3, and blocks 1and 7 are grey white; (d) blocks 1, 3, 4 and 5 are all grey white, andthe block 0 is not pure white.

Process 500 will be described below in connection with the images shownin FIGS. 1A and 1B.

FIG. 1A illustrates an image with a main color of pure white and fewerthan five valid colors. When divided according to the scheme shown inFIG. 4, the color characteristic values of respective image blocks areillustrated below:

666 666 666 666 666 666 666 666 666

According to process 500, at 516, the number of white blocks is 9. Thus,at 518, it is determined whether the number of grey white blocks is 0,the number of pure white blocks is greater than 6, and the number ofbright white blocks is greater than 7. Since the answer is no, it isdetermined whether the number of dark white blocks is 9 and whether thenumber of grey white blocks is greater than 6. In this case, thecondition is true, thus the image is determined to be a non-productimage.

FIG. 1B illustrates an image with a main color of pure white and fewerthan five valid colors. When divided according to the scheme shown inFIG. 4, the color characteristic values of respective image blocks areillustrated below:

777 777 777 777 777 777 777 777 777

According to process 500, it is determined at 514 that the number ofwhite blocks is 9. It is further determined at 518 whether the number ofgrey white blocks is 0, the number of pure white blocks is greater than6 and the number of bright white blocks is greater than 7. In this casethe answer is yes, thus it is determined at 520 whether block 4 isbright white and at least one of blocks 1, 4, and 7 is pure white. Theseconditions are also true, thus the image is deemed to be non-product.

The foregoing solution applies a digital image analysis technique tofirstly extract a color feature of the image and then determine whetherthe image is a non-product image in combination with a predetermineddetermination rule to automatically distinguish a non-product image froma product image among pictures of a product so that the system canprocess them differently. Of course, the determination rule described inthe present embodiment is merely a specific rule derived from real data,those skilled in the art can define various determination rulesaccording to different application demands and the application will notbe limited in this respect.

The technical solution according to the application can be applied tothe stage of uploading a picture from the user and upon detection of thepicture uploaded from the user being a non-product image, the system canreject to accept the picture or feed a message back to the useruploading the non-product image and prompt him to re-upload it tothereby ensure validity of the picture in the system. The foregoingsolution can also be applied prior to data mining to reduce influenceupon the data mining by precluding a non-product image. Also therecognized non-product image can be cleared to save a storage space ofthe system.

FIG. 6 is a block diagram illustrating an embodiment of a systemconfigured to recognize non-product images. System 600 may beimplemented using one or more computing devices such as a personalcomputer, a server computer, a handheld or portable device, a flat paneldevice, a multi-processor system, a microprocessor based system, aset-top box, a programmable consumer electronic device, a network PC, aminicomputer, a large-scale computer, a special purpose device, adistributed computing environment including any of the foregoing systemsor devices, or other hardware/software/firmware combination thatincludes one or more processors, and memory coupled to the processorsand configured to provide the processors with instructions.

In the example shown, system 600 includes a feature value acquisitionunit 810 adapted to obtain color characteristic values of respectivepixels in a source image. It further includes a source image main colordetermination unit 820 adapted to determine a main color of the sourceimage. In some embodiments, the main color determination unit isconfigured to count the number of the pixels corresponding to therespective color characteristic values and make the determination basedon the counting result. Also included is a first recognition unit 830adapted to recognize whether the source image is a non-product imagebased on the color characteristic value of main color of the sourceimage.

FIG. 7 is a block diagram illustrating another embodiment of a systemconfigured to recognize non-product images. Similar to system 600,system 700 also includes feature value acquisition unit 810, main colordetermination unit 820, and first recognition unit 830. It furtherincludes an image conversion unit 800 adapted to convert the sourceimage into a preset format before the characteristic value acquisitionunit 810 acquires the characteristic values and an image blockprocessing unit 840 adapted to divide the source image into severalimage blocks and to determine main colors of the respective imageblocks. In addition it includes a second recognition unit 850 adapted torecognize whether the source image is a non-product image according tothe main colors of the respective image blocks.

The units described above can be implemented as software componentsexecuting on one or more general purpose processors, as hardware such asprogrammable logic devices and/or Application Specific IntegratedCircuits designed to perform certain functions or a combination thereof.In some embodiments, the units can be embodied by a form of softwareproducts which can be stored in a nonvolatile storage medium (such asoptical disk, flash storage device, mobile hard disk, etc.), including anumber of instructions for making a computer device (such as personalcomputers, servers, network equipments, etc.) implement the methodsdescribed in the embodiments of the present invention. The units may beimplemented on a single device or distributed across multiple devices.The functions of the units may be merged into one another or furthersplit into multiple sub-units.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. An image recognition method, comprising:obtaining color characteristic values of pixels in a source image;determining a main color of the source image based at least in part onthe color characteristic values; and determining whether the sourceimage is a non-product image based at least in part on the main color ofthe source image, including: performing a preliminary determination ofwhether the source image is likely to be a non-product image based atleast in part on the main color of the source image; in the event thatit is determined that the source image is likely to be a non-productimage: dividing the source image into a plurality of image blocks;determining main colors of the respective image blocks; and determiningwhether the source image is a non-product image based at least in parton the main colors of the respective image blocks.
 2. The method ofclaim 1, wherein determining the main color of the source image includesdetermining number of pixels corresponding to the color characteristicvalues of pixels in the source image and identifying a color thatcorresponds to the color characteristic value that has the highest pixelcount.
 3. The method of claim 1, wherein determining whether the sourceimage is a non-product image based at least in part on the main colorsof the respective image blocks includes determining whether the sourceimage is a non-product image based at least in part on number of imageblocks with a specific main color.
 4. The method of claim 1, whereindetermining whether the source image is a non-product image based atleast in part on the main colors of the respective image blocks includesdetermining whether the source image is a non-product image based atleast in part on a relative position of an image block with a specificmain color in the source image.
 5. The method of claim 1, whereindividing the source image into a plurality of image blocks comprisesdividing the source image into nine image blocks in a 3×3 layout.
 6. Themethod of claim 1, further comprising converting the source imageaccording to a preset format; and wherein obtaining the colorcharacteristic values of the pixels in the source image comprisesacquiring the color characteristic values of the pixels in the convertedformat.
 7. The method of claim 1, wherein the color characteristicvalues are RGB values.
 8. An image recognition method, comprising:obtaining color characteristic values of pixels in a source image;determining a main color of the source image based at least in part onthe color characteristic values; and determining whether the sourceimage is a non-product image based at least in part on the main color ofthe source image; including: determining whether the main color is blackor white; in the event that the main color is neither black nor white,recognizing the source image as a product image; in the event that themain color is black or white: dividing the source image into nine imageblocks of a 3×3 layout and recognizing whether the source image is anon-product image according to the number of image blocks with aspecific main color.
 9. An image recognition method, comprising:obtaining color characteristic values of pixels in a source image;determining a main color of the source image based at least in part onthe color characteristic values; and determining whether the sourceimage is a non-product image based at least in part on the main color ofthe source image; including: determining whether the main color is blackor white; in the event that the main color is neither black nor white,recognizing the source image as a product image; in the event that themain color is black or white: dividing the source image into nine imageblocks of a 3×3 layout and recognizing whether the source image is anon-product image according to a relative position of an image blockwith a specific main color in the source image.
 10. An image recognitionsystem comprising: one or more processors configured to: obtain colorcharacteristic values of pixels in a source image; determine a maincolor of the source image based at least in part on the colorcharacteristic values; and determine whether the source image is anon-product image based at least in part on the main color of the sourceimage, including to: perform a preliminary determination of whether thesource image is likely to be a non-product image based at least in parton the main color of the source image; in the event that it isdetermined that the source image is likely to be a non-product image,to: divide the source image into a plurality of image blocks; determinemain colors of the respective image blocks; and determine whether thesource image is a non-product image based at least in part on the maincolors of the respective image blocks; and one or more memories coupledto the one or more processors, configured to provide the one or moreprocessors with instructions.
 11. The system of claim 10, whereindetermining the main color of the source image includes determiningnumber of pixels corresponding to the color characteristic values ofpixels in the source image and identifying a color that corresponds tothe color characteristic value that has the highest pixel count.
 12. Thesystem of claim 10, wherein determining whether the source image is anon-product image based at least in part on the main colors of therespective image blocks includes determining whether the source image isa non-product image based at least in part on number of image blockswith a specific main color.
 13. The system of claim 10, whereindetermining whether the source image is a non-product image based atleast in part on the main colors of the respective image blocks includesdetermining whether the source image is a non-product image based atleast in part on a relative position of an image block with a specificmain color in the source image.
 14. The system of claim 10, whereindividing the source image into a plurality of image blocks comprisesdividing the source image into nine image blocks in a 3×3 layout. 15.The system of claim 10, wherein the one or more processors are furtherconfigured to: convert the source image according to a preset format;and wherein obtaining the color characteristic values of the pixels inthe source image comprises acquiring the color characteristic values ofthe pixels in the converted format.
 16. An image recognition systemcomprising: one or more processors configured to: obtain colorcharacteristic values of pixels in a source image; determine a maincolor of the source image based at least in part on the colorcharacteristic values; and determine whether the source image is anon-product image based at least in part on the main color of the sourceimage; wherein: in the event that the main color is neither black norwhite, the one or more processors are further configured to recognizethe source image as a product image; in the event that the main color isneither black nor white, the one or more processors are furtherconfigured to recognize the source image as a product image; in theevent that the main color is black or white, the one or more processorsare further configured to: divide the source image into nine imageblocks of a 3×3 layout; and recognize whether the source image is anon-product image according to the number of image blocks with aspecific main color.
 17. An image recognition system comprising: one ormore processors configured to: obtain color characteristic values ofpixels in a source image; determine a main color of the source imagebased at least in part on the color characteristic values; and determinewhether the source image is a non-product image based at least in parton the main color of the source image; wherein: in the event that themain color is neither black nor white, the one or more processors arefurther configured to recognize the source image as a product image; inthe event that the main color is neither black nor white, the one ormore processors are further configured to recognize the source image asa product image; in the event that the main color is black or white, theone or more processors are further configured to: divide the sourceimage into nine image blocks of a 3×3 layout; and recognize whether thesource image is a non-product image according to a relative position ofan image block with a specific main color in the source image.
 18. Acomputer program product for recognizing image, the computer programproduct being embodied in a non-transitory tangible computer readablestorage medium and comprising computer instructions for: obtaining colorcharacteristic values of pixels in a source image; determining a maincolor of the source image based at least in part on the colorcharacteristic values; and determining whether the source image is anon-product image based at least in part on the main color of the sourceimage, including: performing a preliminary determination of whether thesource image is likely to be a non-product image based at least in parton the main color of the source image; in the event that it isdetermined that the source image is likely to be a non-product image:dividing the source image into a plurality of image blocks; determiningmain colors of the respective image blocks; and determining whether thesource image is a non-product image based at least in part on the maincolors of the respective image blocks.